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I. 


Introduction 


The recent interest within the Internet community in determining 
accurate time from a set of mutually suspicious network clocks has 
been prompted by several occasions in which gross errors were found 
in usually reliable, highly accurate clock servers after seasonal 
thunderstorms which disrupted their primary power supply. To these 
sources of error should be added those due to malfunctioning 
hardware, defective software and operator mistakes, as well as random 
errors in the mechanism used to set and/or synchronize the clocks via 
Internet paths. The results of these errors can range from simple 
disorientation to major disruption, depending upon the operating 
system, when files or messages are incorrectly timestamped or the 
order of critical network transactions is altered. 


This report suggests a stochastic model based on the principles of 
maximum-likelihood estimation, together with algorithms for computing 
a good estimator from a number of time-offset samples measured 
between one or more clocks connected via network links. The model 
provides a rational method for detecting and resolving errors due to 
faulty clocks or excessively noisy links. Included in this report 
are descriptions of certain experiments conducted with Internet hosts 
and ARPANET paths which give an indication of the effectiveness of 
the algorithms. 


Several mechanisms have been specified in the Internet protocol suite 
to record and transmit the time at which an event takes place, 
including the ICMP Timestamp message [6], Time Protocol [7], Daytime 
protocol [8] and IP Timestamp option [9]. A new Network Time 
Protocol [12] has been proposed as well. Additional information on 
network time synchronization can be found in the References at the 
end of this document. Synchronization protocols are described in [3] 
and [12] and synchronization algorithms in [2], [5] and [10]. 
Experimental results on measured roundtrip delays and clock offsets 
in the Internet are discussed in [4] and [11]. A comprehensive 
mathematical treatment of clock synchronization can be found in [1]. 


In [10] the problem of synchronizing a set of mutually suspicious 
clocks is discussed and algorithms offered which maximize in some 
sense the expectation that a correct set of "good" clocks can be 
extracted from the population including also "bad" ones. The 
technique is based upon overlapping, discrete confidence intervals 
which are assigned a-priori. The model assumes the reasonable 
presumption that "bad" clocks display errors far outside these 
confidence intervals, so can be easily identified and discarded from 
the voting process. 
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As apparent from the data summarized in Appendix A, host clocks ina 
real network commonly indicate various degrees of dispersion with 
respect to each other and to a standard-time reference such as a 
radio clock. The sources of dispersion include random errors due to 
observational phenomena and the synchronization mechanism itself, if 
used, as well as systematic errors due to hardware or software 
failure, poor radio reception conditions or operator mistakes. In 
general, it is not possible to accurately quantify whether the 
dispersion of any particular clock qualifies the clock as "good" or 
"bad," except on a statistical basis. Thus, from a practical 
standpoint, a statistical-estimation approach to the problem is 
preferred over a discrete-decision one. 


A basic assumption in this report is that the majority of "good" 
clocks display errors clustered around a zero offset relative to 
standard time, as determined for example from a radio clock, while 
the remaining "bad" clocks display errors distributed randomly over 
the observing interval. The problem is to select the good clocks 
from the bad and to estimate the correction to apply to the local 
clock in order to display the most accurate time. The algorithms 
described in this report attempt to do this using maximum-likelihood 
techniques, which are theory. 


It should be noted that the algorithms discussed in [10] and in this 
report are are basically filtering and smoothing algorithms and can 
result in errors, sometimes gross ones, if the sample distribution 


departs far from a-priori assumptions. Thus, a significant issue in 
the design of these algorithms is robustness in the face of skewed 
sample data sets. The approach in [10] uses theorem-proving to 


justify the robustness of the discrete algorithms presented; 
however, the statistical models in this report are not suited for 
that. The approach taken in this report is based on detailed 
observation and experiments, a summary of which is included as an 
appendix. While this gives an excellent qualitative foundation upon 
which to judge robustness, additional quantitative confidence should 
be developed through the use of statistical mechanics. 


2. Majority-Subset Algorithms 


A stochastic model appropriate to a system of mutually suspicious 
clocks can be constructed as follows. An experiment consists of one 
or more measurements of time differences or offsets between several 
clocks in the network. Usually, but not necessarily, one of the 
clocks is the local clock at the observer and observations are 
conducted with each of several other clocks in the network. The fact 
that some clocks are presumed more accurate or trusted more highly 
than others can be expressed by weighting the measurements 
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accordingly. The result is a set of statistics, including means and 
variances, from which the observer is able to estimate the best time 
at which to set the local clock. 


A maximum-likelihood estimator is a statistic that maximizes the 
probability that a particular outcome of an experiment is due to a 
presumed set of assumptions on the constraints of the experiment. 

For example, if it is assumed that at least k of n observations 
include only samples from a single distribution, then a 
maximum-likelihood estimator for the mean of that distribution might 
be computed as follows: Determine the variance for every k-sample 
subset of the n observations. Then select the subset with smallest 
variance and use its mean as the estimator for the distribution mean. 


For instance, let n be the number of clocks and k be the next largest 
integer in n/2, that is, the minimum majority. A majority subset is 


a subset consisting of k of the n offset measurements. Each of these 
subsets can be represented by a selection of k out of n 
possibilities, with the total number of subsets equal to C(n,k). The 


number of majority subsets is tallied for n from 2 to 20 in Table 1. 


(n,k) C (n,k) (n,k) C (n,k) 
(2,2) 1 (11,6) 462 
(3,2) 3 (12575 792 
(4,3) 4 (13,7) 1716 
(5,3) 10 (14,8) 3003 
(6,4) 15 (15,8) 6435 
(7,4) 35 (16,9) 11440 
(8,5) 56 (17,9) 24310 
(9,5) 126 (18,10) 43758 
(10,6) 210 (19,10) 92378 
(20,11) 167960 


Table 1. C(n,k) for n from 2 to 20 
Obviously, the number of computations required becomes awkward as n 


increases beyond about 10. Representative majority subsets for n = 
3,4,5 are shown in Table 2. 
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Table 2. Majority Subsets for n = 3,4,5 


Choosing n = 5, for example, requires calculation of the mean and 
variance for ten subsets indexed as shown in Table 2. 


A maximum-likelihood algorithm with provision for multiple samples 
and weights might operate as follows: Let n be the number of clocks 
and w(1),w(2),...,w(n) a set of integer weights, with w(i) the weight 
associated with the ith clock. For the ith clock three accumulators 
W(i), X(i) and Y(i) are provided, each initialized to zero. The ith 
clock is polled some number of times, with each reply x causing n(i) 
to be added to W(i), as well as the weighted sample offset n(i)*x 
added to X(i) and its square (n(i)*x)2 added to Y(i). Polling is 
continued for each of the n clocks in turn. 


Next, using a majority-subset table such as shown in Table 2, 


calculate the total weight W = sum(W(i)) and weighted sums X = 
sum(X(i)) and Y = sum(Y(i)*Y(i)) for each i in the jth majority 
subset (row). From W, X and Y calculate the mean m(j) and variance 
s (j): 

m(j) = X/W and s(3) = Y/W - m(3)*m(3) 


When this is complete for all rows, select the row j with the 
smallest s(j) and return the associated mean m(j) as the 
maximum-likelihood estimate of the local-clock offset. 
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34 


Clustering Algorithms 


Another method for developing a maximum-likelihood estimator is 
through the use of clustering algorithms. These algorithms operate 
to associate points in a sample set with clusters on the basis of 
stochastic properties and are most useful when large numbers of 
samples are available. One such algorithm operates on a sample set 
to selectively discard points presumed outside the cluster as 
follows: 


1. Start with a sample set of n observations {x(1),x(2),...,x(n) 
2. Compute the mean of the n observations in the sample set. 


Discard the single sample x(i) with value furthest from the 
mean, leaving n-1 observations in the set. 


3. Continue with step 2 until only a single observation is left, 
at which point declare its value the maximum-likelihood 
estimator. 


This algorithm will usually (but not necessarily) converge to the 
desired result if the majority of observations are the result of 
"good" clocks, which by hypothesis are clustered about zero offset 
relative to the reference clock, with the remainder scattered 
randomly over the observation interval. 


The following Table 3 summarizes the results of this algorithm 
applied to the UDP data shown in Appendix A, which represents the 
measured clock offsets of 163 host clocks in the Internet system. 
These data were assembled using the UDP Time protocol [7], in which 
time is represented to a precision of one second. Each line of the 
table represents the result of step 2 above along with the size of 
the sample set and its (unweighted) mean and variance. The "Discard" 
column shows the value of the observation discarded at that step. 
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Size Mean Var Discard 
163 -210 9.1E+6 -38486 
162 26 172289 3728 
161 3 87727 3658 
160 -20 4280 -566 
150 57 1272 88 

100 -18 247 -44 

50 -4 35 8 

20 -1 0 -2 

19 -1 0 -2 

18 -1 0 -2 

1.7 =l 0 T. 

16 -1 0 -1 

15 =f 0 =f 

14 -1 0 =k 

13 0 0 0 

1 0 0 0 


Table 3. Clustering Algorithm using UDP Time Protocol Data 


In Table 3 only a few of the 163 steps are shown, including those 
near the beginning which illustrate a rapid convergence as the 
relatively few outliers are discarded. The large outlier discarded 
in the first step is almost certainly due to equipment or operator 
failure. The two outliers close to one hour discarded in the next two 
steps are probably simple operator mistakes like entering summer time 
instead of standard time. By the time only 50 samples are left, the 
error has shrunk to about 4 sec and the largest outlier is within 12 
sec of the estimate. By the time only 20 samples are left, the error 
has shrunk to about a second and the variance has vanished for 
practical purposes. 


The following Table 4 summarizes the results of the clustering 
algorithm applied to the ICMP data shown in Appendix A, which 
represents the measured clock offsets of 504 host clocks in the 
Internet system. These data were assembled using ICMP Timestamp 
messages [6], in which time is represented to a precision of one 
millisecond. The columns in Table 4 should be interpreted in the 
same way as in Table 3, except that the data in Table 4 are in 
milliseconds, while the data in Table 3 are in seconds. 
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Size Mean Var Discard 
504 -3.0E+6 3.2E+14 8.6E+7 
500 -3.3E+6 2.9E+14 8.6E+7 
450 -1.6E+6 3.0E+13 -2.5E+7 
400 29450 2.2E+11 3.6E+6 
350 -3291 4.1E+9 -185934 
300 3611 1.6E+9 -95445 
250 2967 6.8E+8 66743 
200 4047 2.3E+8 39288 
150 1717 8.6E+7 21346 
100 803 1.9E+7 10518 
80 1123 8.4E+6 -4863 
60 1119 3.1E+6 4677 

50 502 1.5E+6 -2222 
40 432 728856 2152 

30 84 204651 -987 

20 30 12810 338 

15 28 2446 122 

10 df: 454 49 

8 =2 196 24 

6 -9 23 0 

4 -10 5 -13 

2 -8 0 -8 


Table 4. Clustering Algorithm using ICMP Timestamp Data 


As in Table 3 above, only some of the 504 steps are shown in Table 4. 
The distinguishing feature of the data in Table 4 is that the raw 
data are much more noisy - only some 30 host clocks are closer than 
one second from the reference clock, while half were further than one 
minute and over 100 further than one hour from it. The fact that the 
algorithm converged to within 8 msec of the reference time under 
these conditions should be considered fairly remarkable in view of 
the probability that many of the outliers discarded are almost 
certainly due to defective protocol implementations. Additional 
information on these experiments is presented in Appendix A. 
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4. Application to Time-Synchronization Data 


A variation of the above algorithms can be used to improve the offset 
estimates from a single clock by discarding noise samples produced by 
occasional retransmissions in the network, for example. A set of n 
independent samples is obtained by polling the clock. Then, a 
majority-subset table is used to compute the m(j) and s(j) statistics 
and the maximum-likelihood estimate determined as above. For this 
purpose the majority-subset table could include larger subsets as 
well. In this manner the maximum-likelihood estimates from each of 
several clocks can be determined and used in the algorithm above. 


In order to test the effectiveness of this algorithm, a set of 
experiments was performed using two WIDEBAND/EISN gateways equipped 
with WWVB radio clocks and connected to the ARPANET. These 
experiments were designed to determine the limits of accuracy when 
comparing these clocks via ARPANET paths. One of the gateways 
(ISI-MCON-GW) is located at the Information Sciences Institute near 
Los Angeles, while the other (LL-GW) is located at Lincoln 
Laboratories near Boston. Both gateways consist of PDP11/44 
computers running the EPOS operating system and clock-interface 
boards with oscillators phase-locked to the WWVB clock. 


The clock indications of the WIDEBAND/EISN gateways were compared 
with the DCNet WWVB reference clock using ICMP Timestamp messages, 
which record the individual timestamps with a precision of a 
millisecond. However, the path over the ARPANET between these 
gateways and the measurement host can introduce occasional 
measurement errors as large as several seconds. In principle the 
effect of these errors can be minimized by using a large sample 
population; however, use of the above algorithms allows higher 
accuracies to be obtained with far fewer samples. 


Measurements were made separately with each of the two gateways by 
sending an ICMP Timestamp Request message from the ARPANET address of 
DCN1 to the ARPANET address of the gateway and computing the 
round-trip delay and clock offset from the ICMP Timestamp Reply 
message. This process was continued for 1000 message exchanges, 
which took from seven minutes to several hours, depending on the 
sample interval selected. 


The tables below summarize the results of both the majority-subset 
and clustering algorithms applied to the data from three experiments, 
one with ISI-MCON-GW and two with LL-GW. The ISI-MCON-GW and LL-GW 
(a) experiments were designed to determine the limits of accuracy 
when using a continuous sequence of request/reply volleys, which 
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resulted in over two samples per second. The remaining LL-GW (b) 
experiment was designed to determine the limits of accuracy using a 
much lower rate of about one sample every ten seconds. 


For each of the three experiments two tables are shown, one using the 
majority-subset algorithm and the other the clustering algorithm. The 
two rows of the majority-subset tables show the statistics derived 
both from the raw data and from the filtered data processed by a 
C(5,3) majority-subset algorithm. In all cases the extrema and 
variance are dramatically less for the filtered data than the raw 
data, lending credence to the conjecture that the mean statistic for 
the filtered data is probably a good maximum-likelihood estimator of 
the true offset. 


Mean Var Max Min 
Raw data 637 2.1E+7 32751 -1096 
C (5,3) =15 389 53 -103 


Table 5. ISI-MCON-GW Majority-Subset Algorithm 


Size Mean Var Discard 
1000 637 2.1E+7 32751 
990 313 1.0E+7 32732 
981 15 1.0E+6 32649 
980 -18 2713 -1096 
970 -15 521 -122 
960 -15 433 -97 
940 =15 332 -75 
900 -15 239 26 
800 Si 141 12 
700 -16 87 5 

600 She 54 -31 
500 -16 33 =5 
400 -18 18 -9 
300 -19 7 -12 
200 -19 2 -21 
100 -18 0 -19 

1 -17 0 -17 


Table 6. ISI-MCON-GW Clustering Algorithm 
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32750 
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Size Mean Var Discard 
1000 377 2.1E+7 -32758 
990 315 1.0E+7 32741 
981 18 1.1E+6 32704 
980 -16 16119 -1392 
970 =1:1 5382 554 
960 =21 3338 311 
940 -24 2012 168 
900 -22 1027 -137 
800 -23 430 -72 
700 -23 255 =55 
600 -22 167 -45 
500 -22 109 -40 
400 -21 66 -6 
300 -18 35 -29 
200 -17 15 -23 
100 -19 3 -15 
50 -21 0 -19 
20 -21 0 -21 
10 -20 0 -20 

1 -20 0 -20 


Table 10. LL-GW (b) Clustering Algorithm 


The rows of the clustering tables show the result of selected steps 
in the algorithm as it discards samples furthest from the mean. The 
first twenty steps or so discard samples with gross errors over 30 
seconds. These samples turned out to be due to a defect in the 
timestamping procedure implemented in the WIDEBAND/EISN gateway code 
which caused gross errors in about two percent of the ICMP Timestamp 
Reply messages. These samples were left in the raw data as received 
in order to determine how the algorithms would behave in such extreme 
cases. As apparent from the tables, both the majority-subset and 
clustering algorithms effectively coped with the situation. 


In the statement of the clustering algorithm the terminating 
condition was specified as when only a single sample is left in the 
sample set. However, it is not necessary to proceed that far. For 
instance, it is known from the design of the experiment and the 
reference clocks that accuracies better than about ten milliseconds 
are probably unrealistic. A rough idea of the accuracy of the mean 
is evident from the deviation, computed as the square root of the 
variance. Thus, attempts to continue the algorithm beyond the point 
where the variance drops below 100 or so are probably misguided. 
This occurs when between 500 and 900 samples remain in the sample 
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set, depending upon the particular experiment. Note that in any case 
between 300 and 700 samples fall within ten milliseconds of the final 
estimate, depending on experiment. 


Comparing the majority-subset and clustering algorithms on the basis 
of variance reveals the interesting observation that the result of 
the C(5,3) majority-subset algorithm is equivalent to the clustering 
algorithm when between roughly 900 and 950 samples remain in the 
sample set. This together with the moderately high variance in the 
ISI-MCON-GW and LL-GW (b) cases suggests a C(6,4) or even C(7,4) 
algorithm might yield greater accuracies. 


5. Summary and Conclusions 


The principles of maximum-likelihood estimation are well known and 
widely applied in communication electronics. In this note two 
algorithms using these principles are proposed, one based on 
majority-subset techniques appropriate for cases involving small 
numbers of samples and the other based on clustering techniques 
appropriate for cases involving large numbers of samples. 


The algorithms were tested on raw data collected with Internet hosts 
and gateways over ARPANET paths for the purpose of setting a local 
host clock with respect to a remote reference while maintaining 
accuracies in the order of ten milliseconds. The results demonstrate 
the effectiveness of these algorithms in detecting and discarding 
glitches due to hardware or software failure or operator mistakes. 
They also demonstrate that time synchronization can be maintained 
across the ARPANET in the order of ten milliseconds in spite of 
glitches many times the mean roundtrip delay. 


The results point to the need for an improved time-synchronization 
protocol combining the best features of the ICMP Timestamp message 
[6] and UDP Time protocol [7]. Among the features suggested for this 
protocol are the following: 


1. The protocol should be based on UDP, which provides the 
flexibility to handle simultaneous, multiplexed queries and 
responses. 


2. The message format should be based on the ICMP Timestamp 
message format, which provides the arrival and departure times 
at the server and allows the client to calculate the roundtrip 
delay and offset accurately. 
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3. The data format should be based on the UDP Time format, which 
specifies 32-bit time in seconds since 1 January 1900, but 
extended additional bits for the fractional part of a second. 


4. Provisions to specify the expected accuracy should be included 
along with information about the reference clock or 
synchronizing mechanism, as well as the expected drift rate 
and the last time the clock was set or synchronized. 


The next step should be formulating an appropriate protocol with the 
above features, together with implementation and test in the Internet 
environment. Future development should result in a distributed, 
symmetric protocol, similar perhaps to those described in [1], for 
distributing highly reliable timekeeping information using a 
hierarchical set of trusted clocks. 
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Appendix A. 
Experimental Determination of Internet Host Clock Accuracies 


Following is a summary of the results of three experiments designed 
to reveal the accuracies of various Internet host clocks. The first 
experiment uses the UDP Time protocol, which is limited in precision 
to one second, while the second uses the ICMP Timestamp message, 
which extends the precision to one millisecond. In the third 
experiment the results indicated by UDP and ICMP are compared. In 
the UDP Time protocol time is indicated as a 32-bit field in seconds 
past 0000 UT on 1 January 1900, while in the ICMP Timestamp message 
time is indicated as a 32-bit field in milliseconds past 0000 UT of 
each day. 


All experiments described herein were conducted from Internet host 
DCN6.ARPA, which is normally synchronized to a WWV radio clock. In 
order to improve accuracy during the experiments, the DCN6.ARPA host 
was resynchronized to a WWVB radio clock. As the result of several 
experiments with other hosts equipped with WWVB and WWV radio clocks 
and GOES satellite clocks, it is estimated that the maximum 
measurement error in the following experiments is less than about 30 
msec relative to standard NBS time determined at the Boulder/Fort 
Collins transmitting sites. 


A1. UDP Time Protocol Experiment 


In the first experiment four UDP Time protocol requests were sent 
at about three-second intervals to each of the 1775 hosts listed 
in the NIC Internet host table. A total of 555 samples were 
received from 163 hosts and compared with a local reference based 
on a WWVB radio clock, which is known to be accurate to within a 
few milliseconds. Not all of these hosts were listed as 
supporting the UDP Time protocol in the NIC Internet host table, 
while some that were listed as supporting this protocol either 
failed to respond or responded with various error messages. 
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In the following table "Host" 


and "Count" the number of replies received. 
in seconds, 


represent the time offset, 


local clock to agree with the host cited. 


(reference) 


September 1 


is the canonical name of the host 


The remaining data 


necessary to correct the 


The "Max 


and "Min" represent the maximum and minimum of these offsets, 
while "Mean" represents the mean value and "Var" the variance, 
rounded to the nearest second. 
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OLUMBIA.ARPA 


SSSSSSESSSSESESES 


C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 
C 


CYPRESS .ARPA 


U-ARPA.CS.CORNELL. EDU 


A APUN ARA QQ 0ds ds 0D 0 ds ds ds ds ds ds 4 Y 0 0 Hs 0 Y Ys 0 da N WHE BPA 


Min Mean Var 
-12 =11 0 
-12 -11 0 
-12 -11 0 
22 22 0 
87 87 0 
FL 71 0 
-1 =i 0 
-95 -94 0 
5 5 0 
-37 >37 0 
-43 -42 0 
-31 -30 0 
-43 -42 0 
oT -36 0 
-45 -44 0 
-44 -44 0 
-31 =31 0 
-75 -74 0 
-40 -39 0 
-50 -49 0 
-132 S131 0 
=37 -36 0 
-45 -44 0 
-66 -66 0 
-41 -41 0 
-45 -44 0 
-28 -27 0 
-19 -18 0 
-50 -49 0 
=33 =33 0 
42 42 0 
-49 -48 0 
-41 -40 0 
-65 -65 0 
8 8 0 
3 4 0 
1 1 0 
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DCN1.ARPA 4 0 0 0 0 
DCN5.ARPA 4 0 0 0 0 
DCN6.ARPA 4 0 0 0 0 
DCN7.ARPA 4 -1 -1 0 0 
DCN9.ARPA 4 =3 =3 =3 0 
DEVVAX.TN.CORNELL. EDU 2 3659 3658 3658 0 
ENEEVAX.ARPA 4 73 72 72 0 
FORD-WDL1.ARPA 4 -59 -60 259 0 
FORD1.ARPA 4 0 0 0 0 
GUENEVERE . PURDUE . EDU 3 1 0 0 0 
GVAX.CS.CORNELL.EDU 4 -18 -18 -18 0 
IM4U.ARPA 4 -6 -6 -6 0 
IPTO-FAX.ARPA 4 0 0 0 0 
KANKIN.ARPA 4 =3 -4 =3 0 
MERLIN.PURDUE.EDU 2 3 3 3 0 
MIT-ACHILLES.ARPA 4 16 16 16 0 
MIT-AGAMEMNON.ARPA 4 -63 -64 -63 0 
MIT-ANDROMACHE. ARPA 4 -28 -28 -28 0 
MIT-APHRODITE.ARPA 4 =1 -8 -7 0 
MIT-APOLLO.ARPA 4 -8 -9 -8 0 
MIT-ARES.ARPA 4 -25 -26 =25 0 
MIT-ARTEMIS.ARPA 4 -34 89 -34 0 
MIT-ATHENA.ARPA 4 -37 -37 -37 0 
MIT-ATLAS.ARPA 4 -26 -26 -26 0 
MIT-CASTOR.ARPA 4 =35 -35 -35 0 
MIT-DAFFY-DUCK.ARPA 2 -72 HS -72 0 
MIT-DEMETER.ARPA 4 -28 -29 -28 0 
MIT-GOLDILOCKS.ARPA 1 -20 -20 -20 0 
MIT-HECTOR.ARPA 4 +23 -24 =23 0 
MIT-HELEN.ARPA 4 6 5 5 0 
MIT-HERA.ARPA 4 -34 =35 -34 0 
MIT-HERACLES.ARPA 4 -36 -36 -36 0 
MIT-JASON.ARPA 4 -36 -37 -36 0 
MIT-MENELAUS. ARPA 4 -32 =33 -32 0 
MIT-MULTICS.ARPA 3 25 23 24 0 
MIT-ODYSSEUS.ARPA 4 20 19 19 0 
MIT-ORPHEUS.ARPA 4 -34 =35 -34 0 
MIT-PARIS.ARPA 4 -35 -35 -35 0 
MIT-POSEIDON.ARPA 4 239 -41 -40 0 
MIT-PRIAM.ARPA 4 -24 =25 -24 0 
MIT-REAGAN.ARPA 4 115 115 115 0 
MIT-THESEUS.ARPA 4 -43 -44 -43 0 
MIT-TRILLIAN.ARPA 4 -38 -39 -38 0 
MIT-TWEETY-PIE.ARPA 3 106 105 105 0 
MIT-ZERMATT.ARPA 4 ES -76 10 0 
MIT-ZEUS.ARPA 4 =31 -39 -38 0 
MOL.ARPA 2 -3 -3 -3 0 
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MUNGO. THINK .COM 4 =1 =1 =1 0 
NETWOLF . ARPA 4 158 157 157 0 
ORBIT.ARPA 3 -4 9 -4 0 
OSLO-VAX.ARPA 3 3729 3727 3728 1 
PATCH.ARPA I 18 18 18 0 
RADC-MULTICS.ARPA 4 -14 =L5 -14 0 
RICE-ZETA.ARPA 1 -31 -31 -31 0 
RICE.ARPA 1 7 7 7 0 
ROCHESTER.ARPA 4 -18 -18 -18 0 
ROCK. THINK.COM 4 2 2 2 0 
SCRC-QUABBIN.ARPA 4 -100 -100 -100 0 
SCRC-RIVERSIDE.ARPA 4 -128 -128 -128 0 
SCRC-STONY-BROOK.ARPA 4 -100 -100 -100 0 
SCRC-VALLECITO. ARPA 4 =37 =97 =37 0 
SCRC-YUKON.ARPA 4 -65 -65 -65 0 
SEBASTIAN. THINK.COM 4 -14 =15 =14 0 
SEISMO.CSS.GOV 3 =I =T 0 0 
SRI-BISHOP.ARPA 4 -40 -41 -40 0 
SRI-DARWIN.ARPA 2 -29 -30 -29 0 
SRI-HUXLEY.ARPA 2 -28 -29 -28 0 
SRI-KIOWA.ARPA 4 =29 -30 =29 0 
SRI-LASSEN.ARPA 3 =LI -12 =LI 0 
SRI-MENDEL.ARPA 4 74 73 73 0 
SRI-PINCUSHION.ARPA 4 -50 SoL -50 0 
SRI-RITTER.ARPA 4 =23 -24 =23 0 
SRI-TIOGA.ARPA 4 127 127 127 0 
SRI-UNICORN.ARPA 4 -38486 -38486 -38486 0 
SRI-WHITNEY.ARPA 4 -24 -24 -24 0 
SRI-YOSEMITE.ARPA 4 -26 -27 -26 0 
SU-AIMVAX.ARPA 2 -54 =59 -54 0 
SU-COYOTE.ARPA 1 14 14 14 0 
SU-CSLI.ARPA 4 SI =I =T 0 
SU-PSYCH.ARPA 1 -52 -52 -52 0 
SU-SAFE.ARPA 1 -60 -60 -60 0 
SU-SIERRA.ARPA 4 S993 =53 -53 0 
SU-SUSHI.ARPA 4 -105 -106 -105 0 
SU-WHITNEY.ARPA 2 -14 -14 -14 0 
TESLA.EE.CORNELL.EDU 3 =2 =3 =2 0 
THORLAC . THINK .COM 4 -20 -20 -20 0 
TRANTOR.ARPA 4 4 3 3 0 
TZEC.ARPA 4 -6 -7 -6 0 
UBALDO. THINK .COM 4 =1:3 =13 =13 0 
UCI-CIP.ARPA 2 -566 -567 -566 0 
UCI-CIP2.ARPA 2 Ho -175 S175 0 
UCI-CIP3.ARPA 2 -89 -90 -89 0 
UCI-CIP4.ARPA 2 zol SSL Sol 0 
UCI-CIP5.ARPA 2 -26 -28 -27 1 
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UCI-ICSA.ARPA 2 -24 -24 -24 0 
UCI-ICSC.ARPA 1 0 0 0 0 
UCI-ICSD.ARPA t -24 -24 -24 0 
UCI-ICSE.ARPA 1 -10 -10 -10 0 
UDEL-DEWEY.ARPA I 88 88 88 0 
UDEL-MICRO.ARPA 2 64 64 64 0 
UIUC.ARPA 4 105 103 104 0 
UIUCDCSB.ARPA 4 65 65 65 0 
UMD1.ARPA 4 0 0 0 0 
UMICH1.ARPA 4 SI SI 0 0 
UO.ARPA 4 -2 =8 -2 0 
USC-ISI.ARPA 4 -45 -45 -45 0 
USC-ISIC.ARPA 4 28 26 27 0 
USC-ISID.ARPA 4 26 25 25 0 
USC-ISIE.ARPA 4 =53 -54 -53 0 
USC-ISIF.ARPA 4 -29 -29 -29 0 
USGS2-MULTICS.ARPA 3 75 74 74 0 
UT-ALAMO.ARPA 4 22 22 22 0 
UT-BARKLEY.ARPA 4 57 56 56 0 
UT-EMIL.ARPA 4 29 28 28 0 
UT-GOTTLOB.ARPA 4 42 41 41 0 
UT-HASKELL.ARPA 4 6 6 6 0 
UT-JACQUES.ARPA 4 pak 20 20 0 
UT-SALLY.ARPA 3 1 0 0 0 
VALENTINE. THINK.COM 4 -10 11 -10 0 
WENCESLAS . THINK.COM 4 -2 =3 -2 0 
XAVIER. THINK.COM 4 -14 -14 -14 0 
XEROX.ARPA 4 0 0 0 0 
YAXKIN.ARPA 3 -4 -5 -4 0 
YON. THINK.COM 4 -11 -12 -11 0 
ZAPHOD . PURDUE . EDU 4 =230 =231 -230 0 
ZOTZ.ARPA 4 E7 16 16 0 


Table Al. UDP Host Clock Offsets for Various Internet Hosts 


The above list includes several host clocks known to be 
synchronized to various radio clocks, including DCN1.ARPA (WWVB), 
DCN6.ARPA (WWV) and FORD1.ARPA (GOES). Under normal radio 
receiving conditions these hosts should be accurate to well within 
a second relative to NBS standard time. Certain other host clocks 
are synchronized to one of these hosts using protocols described 
in RFC-891, including DCN5.ARPA, DCN7.ARPA and UMD1.ARPA 
(synchronized to DCN1.ARPA) and UMICH1.ARPA (synchronized to 
FORD1.ARPA). It is highly likely, but not confirmed, that several 
other hosts with low offsets derive local time from one of these 
hosts or from other radio clocks. 
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The raw statistics computed from the weighted data indicate a mean 
of -261 sec, together with a maximum of 3729 sec and a minimum of 
-38486 sec. Obviously, setting a local clock on the basis of 
these statistics alone would result in a gross error. 


A closer look at the distribution of the data reveals some 
interesting features. Table A2 is a histogram showing the 
distribution within a few seconds of reference time. In this and 
following tables, "Offset" is in seconds and indicates the 
lower-valued corner of the histogram bin, which extends to the 
next higher value, while "Count" indicates the number of samples 
falling in that bin. 


Offset Count Offset Count 
O sec 13 (continued) 
1 al -1 3 
2 T -2 3 
3 2 =3 3 

4 1 -4 2 

5 2 -5 0 

6 1 -6 2 

7 1 -7 1 

8 1 -8 t 

9 0 -9 0 
> 9 30 < -9 95 


Table A2. Offset Distribution < 9 sec 


A total of 16 of the 163 host clocks are within a second in 
accuracy, while a total of 125 are off more than ten seconds. It 
is considered highly likely that most of the 16 host clocks within 
a second in offset are synchronized directly or indirectly to a 
radio clock. Table A3 is a histogram showing the distribution over 
a larger scale. 
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Offset Count Offset Count 
0 sec 35 (continued) 
30 3 -30 50 
60 8 -60 42 
90 3 =90 8 
120 1 -120 4 
150 1 -150 2 
180 0 -180 1 
210 0 -210 0 
240 0 -240 1 
270 0 -270 0 
> 270 2 < -270 2 


Table A3. Offset Distribution < 270 sec 


A total of 138 of the 163 host clocks are within a minute in 
accuracy, while a total of four host clocks are off more than 4.5 
minutes. It is considered likely that most host clocks, with the 
exception of the 16 identified above as probably synchronized to a 
radio clock, are set manually by an operator. Inspection of the 
raw data shows some hosts to be very far off; for instance, 
SRI-UNICORN.ARPA is off more than ten hours. Note the interesting 
skew in the data, which show that most host clocks are set slow 
relative to standard time. 


ICMP Timestamp Messages Experiment 


The the second experiment four ICMP Timestamp messages were sent 
at about three-second intervals to each of the 1775 hosts and 110 
gateways listed in the NIC Internet host table. A total of 1910 
samples were received from 504 hosts and gateways and compared 
with a local reference based on a WWVB radio clock, which is known 
to be accurate to within a few milliseconds. Support for the ICMP 
Timestamp messages is optional in the DoD Internet protocol suite, 
so it is not surprising that most hosts and gateways do not 
support it. Moreover, bugs are known to exist in several widely 
distributed implementations of this feature. The situation proved 
an interesting and useful robustness test for the clustering 
algorithm described in the main body of this note. 


While the complete table of ICMP offsets by host is too large to 
reproduce here, the following Tables A4 through A7 show the 
interesting characteristics of the distribution. The raw 
statistics computed from the weighted data indicate a mean of 
-2.8E+6 msec, together with a maximum of 8.6E+7 msec and a minimum 
of -8.6E+7 msec. Setting a local clock on the basis of these 
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statistics alone would be ridiculous; however, as described in the 
main body of this note, use of the clustering algorithm improves 
the estimate to within 8 msec of the correct value. The apparent 
improvement of about six orders in magnitude is so remarkable as 
to require a closer look at the distributions. 


The reasons for the remarkable success of the clustering algorithm 
are apparent from closer examination of the sequence of histograms 
shown in Tables A4 through A7. Table A4 shows the distribution in 
the scale of hours, from which it is evident that 80 percent of 
the samples lie in a one-hour band either side of zero offset; 
but, strangely enough, there is a significant dispersion in 
samples outside of this band, especially in the negative region. 
It is almost certain that most or all of the latter samples 
represent defective ICMP Timestamp implementations. Note that 
invalid timestamps and those with the high-order bit set 
(indicating unknown or nonstandard time) have already been 
excluded from these data. 


Offset Count Offset Count 
0 hr 204 (continued) 
1 10 -1 194 
2 0 -2 0 

3 0 =3. 2 

4 0 -4 17 
5 0 -5 10 
6 0 -6 E 

7 0 -7 22 
8 0 -8 20 
9 0 =9 0 

> 9 0 < -9 LS 


Table A4. ICMP Offset Distribution < 9 hours 


Table A5 shows the distribution compressed to the range of 4.5 
minutes. About half of the 370 samples remaining after the 
outliers beyond 4.5 minutes are excluded lie in the band 30 
seconds either side of zero offset, with a gradual tapering off to 
the limits of the table. This type of distribution would be 
expected in the case of host clocks set manually by an operator. 
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Offset Count Offset Count 
0 sec TtT (continued) 
30 25 =30 80 
60 26 -60 28 
90 13 =90 18 
120 F -120 19 
150 5 -150 9 
180 3 -180 10 
210 3 -210 6 
240 1: -240 2 
270 2 -270 2 

> 270 29 < -270 105 


Table A5. ICMP Offset Distribution < 270 sec 


Table A6 shows the distribution compressed to the range of 27 
seconds. About 29 percent of the 188 samples remaining after the 
outliers beyond 27 seconds are excluded lie in the band 3 seconds 
either side of zero offset, with a gradual but less pronounced 
tapering off to the limits of the table. This type of 
distribution is consistent with a transition region in which some 
clocks are set manually and some by some kind of protocol 
interaction with a reference clock. A fair number of the clocks 
showing offsets in the 3-27 second range have probably been set 
using the UDP Time protocol at some time in the past, but have 
wandered away as the result of local-oscillator drifts. 


Offset Count Offset Count 
0 sec 32 (continued) 
3 15 -3 22 
6 9 -6 12 
9 6 -9 8 
12 13 -12 8 
LS 5 -15 5 
18 8 -18 9 
21 8 -21 7 
24 9 -24 3 
27 6 -27 3 

> 27 114 < -27 202 


Table A6. ICMP Offset Distribution < 27 sec 
Finally, Table A7 shows the distribution compressed to the range 


of 0.9 second. Only 30 of the original 504 samples have survived 
and only 12 of these are within a band 0.1 seconds either side of 
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zero offset. The latter include those clocks continuously 
synchronized to a radio clock, such as the DCNet clocks, some 
FORDnet and UMDnet clocks and certain others. 


Offset Count Offset Count 
0 sec 6 (continued) 
1 3 Sai 6 
2 1 EZ 3 
3 1 Sa 0 
4 0 -.4 0 
5 1 =g 2 
6 0 56 0 
7 1 Sa 0 
8 4 -.8 2 

.9 0 +29 0 
> .9 208 < -.9 266 


Table A7. ICMP Offset Distribution < .9 sec 


The most important observation that can be made about the above 
histograms is the pronounced central tendency in all of them, in 
spite of the scale varying over six orders of magnitude. Thus, a 
clustering algorithm which operates to discard outliers from the 
mean will reliably converge on a maximum-likelihood estimate close 
to the actual value. 


Comparison of UDP and ICMP Time 


The third experiment was designed to assess the accuracies 
produced by the various host implementations of the UDP Time 
protocol and ICMP Timestamp messages. For each of the hosts 
responding to the UDP Time protocol in the first experiment a 
separate test was conducted using both UDP and ICMP in the same 
test, so as to minimize the effect of clock drift. Of the 162 
hosts responding to UDP requests, 45 also responded to ICMP 
requests with apparently correct time, but the remainder either 
responded with unknown or nonstandard ICMP time (29) or failed to 
respond to ICMP requests at all (88). 


Table A8 shows both the UDP time (seconds) and ICMP time 
(milliseconds) returned by each of the 45 hosts responding to both 
UDP and ICMP requests. The data are ordered first by indicated 
UDP offset and then by indicated ICMP offset. The seven hosts at 
the top of the table are continuously synchronized, directly or 
indirectly to a radio clock, as described earlier under the first 
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but not confirmed, that those hosts 


below showing discrepancies of a second or less are synchronized 
on occasion to one of these hosts. 
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ICMP time 


UMD1.ARPA 
UMICH1.ARPA 
FORD1.ARPA 
TESLA.EE.CORNELL.EDU 
SEISMO.CSS.GOV 
UT-SALLY.ARPA 
CU-ARPA.CS.CORNELL. EDU 
UCI-ICSE.ARPA 
UCI-ICSC.ARPA 
DCN9.ARPA 
TRANTOR.ARPA 
COLUMBIA.ARPA 
GVAX.CS.CORNELL. EDU 
UCI-CIP5.ARPA 
RADC-MULTICS.ARPA 
SU-WHITNEY.ARPA 
UCI-ICSD.ARPA 
SU-COYOTE.ARPA 
MIT-MULTICS.ARPA 
BBNA.ARPA 
UCI-ICSA.ARPA 
ROCHESTER.ARPA 
SU-AIMVAX.ARPA 
UCI-CIP4.ARPA 
SU-SAFE.ARPA 
SU-PSYCH.ARPA 
UDEL-MICRO.ARPA 
UIUCDCSB.ARPA 
BELLCORE-CS-GW.ARPA 
USGS2-MULTICS.ARPA 
BBNG.ARPA 
UDEL-DEWEY . ARPA 
UCI-CIP3.ARPA 
UIUC.ARPA 
UCI-CIP2.ARPA 
UCI-CIP.ARPA 
OSLO-VAX.ARPA 
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-102 
105 

-185 
-576 
3738 


132 

174 
-240 
-514 
-1896 
2000 
-6610 
10232 
12402 
-11988 
-17450 
-16600 
17480 
-20045 
21642 
28265 
-34199 
-36804 
-41542 
-49575 
-57060 
-59212 
-58421 
63214 
63865 
71402 
77018 
81439 
89283 
-102148 
105843 
-185250 
-576386 
3739395 
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DEVVAX.TN.CORNELL. EDU 3657 3657026 
PATCH.ARPA -86380 20411 
IPTO-FAX.ARPA -86402 -1693 
NETWOLF . ARPA 10651435 -62164450 


Table A8. Comparison of UDP and ICMP Host Clock Offsets 


Allowing for various degrees of truncation and roundoff abuse in 
the various implementations, discrepancies of up to a second could 
be expected between UDP and ICMP time. While the results for most 
hosts confirm this, some discrepancies are far greater, which may 
indicate defective implementations, excessive swapping delays and 
so forth. For instance, the ICMP time indicated by UCI-CIP5.ARPA 
is almost 2.5 seconds less than the UDP time. 


Even though the UDP and ICMP times indicated by OSLO-VAX.ARPA and 
DEVVAX.TN.CORNELL.EDU agree within nominals, the fact that they 
are within a couple of minutes or so of exactly one hour early 
(3600 seconds) lends weight to the conclusion they were 
incorrectly set, probably by an operator who confused local summer 
and standard time. 


Something is clearly broken in the case of PATCH.ARPA, 
IPTO-FAX.ARPA and NETWOLF .ARPA. Investigation of the PATCH.ARPA 
and IPTO-FAX.ARPA reveals that these hosts were set by hand 
accidently one day late (-86400 seconds), but otherwise close to 
the correct time-of-day. Since the ICMP time rolls over at 2400 
UT, the ICMP offset was within nominals. No explanation is 
available for the obviously defective UDP and ICMP times indicated 
by NETWOLF.ARPA, although it was operating within nominals at 
least in the first experiment. 
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